System and method of network operating system containers

ABSTRACT

A method and apparatus of a network element that processes control plane data in a network element is described. In an exemplary embodiment, the device receives control plane data with a network element operating system, where at least a functionality of the network element operating system is executing in a container of the network element. In addition, the network element includes a data plane with a plurality of hardware tables and the host operating system. Furthermore, the network element processes the control plane data with the network element operating system. The network element additionally updates at least one of the plurality of hardware tables with the process control plane data using the network element operating system.

RELATED APPLICATIONS

Applicant claims the benefit of priority of prior, co-pendingprovisional application Ser. No. 62/465,121, filed Feb. 28, 2017, theentirety of which is incorporated by reference.

FIELD OF INVENTION

This invention relates generally to data networking, and moreparticularly, to executing network operation system functions inoperating system containers.

BACKGROUND OF THE INVENTION

A network element can include two different planes that are used toprocess network traffic: a control plane; and a data plane that includesone or more hardware forwarding engines. The data plane receives,processes, and forwards network traffic using various configuration data(e.g., forwarding, security, quality of service (QoS), and other networktraffic processing information). The control plane controls theseprocessing functions of the data plane by configuring the data plane,managing data collected by the data plane, monitoring the data plane,and other management functions. The functions for each of the data andcontrol planes are programmed by a software image, the network elementoperating system, that is stored on the network element. When thenetwork element boots up, the software image is loaded and is used toprogram the data plane and control plane.

SUMMARY OF THE DESCRIPTION

A method and apparatus of a network element that processes control planedata in a network element is described. In an exemplary embodiment, thedevice receives control plane data with a network element operatingsystem, where at least a functionality of the network element operatingsystem is executing in a container. In addition, the network elementincludes a data plane, with a plurality of hardware tables, and the hostoperating system. Furthermore, the network element processes the controlplane data with the network element operating system. The networkelement additionally updates at least one of the plurality of hardwaretables with the process control plane data using the network elementoperating system.

In another embodiment, a network element receives control plane datawith at least one of a plurality of processes of a network elementoperating system of the network element, wherein the plurality ofprocesses is executing in a plurality of containers. In addition, thenetwork element includes a data plane with a plurality of hardwaretables. The network element further processes the control plane datausing at least one of the plurality of processes. The network elementadditionally updates at least one of the plurality of hardware tableswith the process control plane data with the at least one of theplurality of processes.

In one embodiment, a network element hitlessly upgrades a networkelement operating system of a network element. In this embodiment, thenetwork element receives a second image for the network elementoperating system, where a first image of the network element operatingsystem is executing as a first set of processes in a first container andthe first set of processes manages the plurality of hardware tables forthe network element. The network element further instantiates a secondcontainer for the second image. In addition, the network element startsa second set of processes using at least the second image in the secondcontainer. The network element additionally synchronizes state databetween the first set of processes and the second set of processes.Furthermore, the network element sets the second set of processes asmanaging the plurality of hardware tables, and deletes the firstcontainer.

In another embodiment, the network element receives a second image forcomponent of the network element operating system, where a first imageof the network element operating system component is executing as afirst set of processes in a first container. In addition, othercomponents of the network element operating system are executing asthird set of processes and at least one other container. The networkelement further instantiates a second container for the second image andstarts a second set of processes using at least the second image in thesecond container. The network element additionally synchronizes statedata between the first set of processes and the second set of processes.In addition, the network element sets the second set of processes ismanaging the functionally of the component and deletes the firstcontainer.

In a further embodiment, a network element installs a device driver usedto manage hardware of the network element. In this embodiment, thenetwork element detects, with a network element operating system, thehardware of a data plane of the network element, where at least onecomponent of the network element operating system is executing in afirst container as a first set of processes and a host operating systeminstantiates the first container. The network element further determinesa device driver for the hardware and installs the device driver in akernel of the host operating system. The network element additionallymanages the data, with the network element operating system, using thedevice driver.

In another embodiment, a device simulates a plurality of networkelements. In one embodiment, the device receives network topologyinformation for the plurality of simulated network elements. The devicefurther instantiates, with a host operating system of the device, acontainer for each of the plurality of simulated network elements. Thedevice additionally configures a set of processes for each of theplurality of containers, where each of the set of processes simulates atleast one of the plurality of simulated network elements. The pluralityof set of processes further implements a network topology represented bythe network topology information. The device performs a test of thenetwork topology and saves the results of the test.

Other methods and apparatuses are also described.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the Figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram of one embodiment of a network element thatincludes a control plane and multiple data planes, with the data planecommunicating data to the control plane.

FIG. 2 is a block diagram of one embodiment of a network element with anetwork element operating system executing within a container.

FIGS. 3A-B are flow diagrams of embodiments of a process to instantiatea container for a network element operating system and to processcontrol plane data using network operating system process(es) withinthis container.

FIG. 4 is block diagram of one embodiment of a network element with anetwork element operating system executing within multiple containers.

FIGS. 5A-B are flow diagrams of embodiments of a process to instantiatemultiple containers for a network element operating system and toprocess control plane data using network operating system process(es)within these containers.

FIG. 6 is a block diagram of one embodiment of a network elementperforming a hitless upgrade for a network element operating systemexecuting within a container of a host operating system.

FIG. 7 is a flow diagrams of one embodiment of a process to perform ahitless upgrade of a network element operating system executing within acontainer of a host operating system.

FIG. 8 is a block diagram of one embodiment of performing a hitlessupgrade for a network element operating system function executing withinone of multiple containers of a host operating system.

FIG. 9 is a flow diagrams of one embodiment of a process to perform ahitless upgrade for a network element operating system functionexecuting within one of multiple containers of a host operating system.

FIG. 10 is a block diagram of one embodiment of a network element thatperforms a device driver installation into a kernel of a host operatingsystem by a network operating system executing in a container.

FIG. 11 is a block diagram of one embodiment of a network element thatperforms a device driver installation into a kernel of a host operatingsystem by a network operating system functions executing in multiplecontainers.

FIG. 12 is a flow diagram of one embodiment of a process to perform adevice driver installation into a kernel of a host operating system by anetwork operating system executing in a container.

FIG. 13 is a block diagram of one embodiment of a topology of networkelements.

FIG. 14 is a block diagram of one embodiment of a device simulatingmultiple network elements in multiple containers.

FIG. 15 is a flow diagram of one embodiment of a process to simulatemultiple network elements in multiple containers.

FIG. 16 illustrates one example of a typical computer system, which maybe used in conjunction with the embodiments described herein.

FIG. 17 is a block diagram of one embodiment of an exemplary networkelement that instantiates a container in a network element.

DETAILED DESCRIPTION

A method and apparatus of a network element that processes control planedata in a network element is described. In the following description,numerous specific details are set forth to provide thorough explanationof embodiments of the present invention. It will be apparent, however,to one skilled in the art, that embodiments of the present invention maybe practiced without these specific details. In other instances,well-known components, structures, and techniques have not been shown indetail in order not to obscure the understanding of this description.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

In the following description and claims, the terms “coupled” and“connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended as synonyms for each other.“Coupled” is used to indicate that two or more elements, which may ormay not be in direct physical or electrical contact with each other,co-operate or interact with each other. “Connected” is used to indicatethe establishment of communication between two or more elements that arecoupled with each other.

The processes depicted in the figures that follow, are performed byprocessing logic that comprises hardware (e.g., circuitry, dedicatedlogic, etc.), software (such as is run on a general-purpose computersystem or a dedicated machine), or a combination of both. Although theprocesses are described below in terms of some sequential operations, itshould be appreciated that some of the operations described may beperformed in different order. Moreover, some operations may be performedin parallel rather than sequentially.

The terms “server,” “client,” and “device” are intended to refergenerally to data processing systems rather than specifically to aparticular form factor for the server, client, and/or device.

A method and apparatus of a network element that processes control planedata in a network element is described. In one embodiment, the networkelement includes a control plane and a data plane that is used toprocess incoming data by the network element. In this embodiment, thenetwork element includes a host operating system that can be used toinstantiate one or more containers, where the containers are used toexecute one or more network element operating system processes.Alternatively, the containers can be each instantiated by anothercontainer or another mechanism. These network element operating systemprocesses, in turn, process any incoming data that is identified ascontrol plane data and program hardware tables with associated hardwaretable updates. In this embodiment, the network element operating system,which executes in the one or more containers, controls the processing ofthe control plane data. Therefore, the network element operating systemcontrols and manages the operation of the network element, even thoughthe network element includes a host operating system that may be from adifferent manufacturer than the one of the network element operatingsystem.

By having the network element operating system execute in a container,the network element operating system acts as an application of the hostoperating system. For example and in one embodiment, there can beprocesses for quality of service functions, access control listsmanagement (or other types of security), policy service, fan agent,light emitting diode agent, temperature sensor agent, database service,management service(s), processes to support networking protocols (e.g.spanning tree protocol (STP), routing protocols (e.g. such as routinginformation protocol (RIP), border gateway protocol (BGP), open shortestpath first (OSPF) protocol, intermediate system-intermediate system(IS-IS) protocol, interior gateway routing protocol (IGRP), enhancedIGRP (EIGRP), protocol independent multicast (PIM), distance vectormulticast routing protocol (DVMRP), and any/or other type or unicast ormulticast routing protocol), Multiprotocol Label Switching (MPLS),and/or other types of networking protocols), network flow managementapplications (e.g., openflow, directflow), process manager, and/or othertypes of processes for other types of functionality of the networkelement.

In another embodiment, the network element operating system ispartitioned into multiple containers, where each of the containers isused to execute one or more processes for different functionalitiesand/or components of the network element operating system. For exampleand in one embodiment, there can be multiple containers and processesfor the functions listed above.

Because the network element operating system can be containerized intoone or more different containers on top of a host operating system,upgrading of the network element operating system, either in whole or inpart, can be performed hitlessly. In one embodiment, if the networkelement operating system is executed in one container, a new containercan be instantiated that is used to perform a hitless upgrade to thenetwork element operating system. A hitless upgrade can be performed byinstantiating a new container for the network element operating system,instantiate the one or more processes for this container, synchronizinga state between the current network element operating system processesand the new network element operating system processes in the newcontainer, and switching over control from the current network elementoperating system processes to the network element operating systemprocesses in the new container.

If the network element operating system is executing in multiplecontainers for different components of the network element operatingsystem, one or more of these components can be upgraded withoutaffecting the execution of processes in other containers. A hitlessupgrade can be performed for the network element operating systemcomponent by instantiating a new container for this component,instantiate the one or more processes of the component for thiscontainer, synchronizing a state between the current component processesand the new component processes in the new container, and switching overcontrol from the current component processes to the component in the newcontainer. Alternatively, the new component process can rebuild a statebased on at least the state of the old component, where control from thecurrent component is switched over to the new component in the newcontainer.

In one embodiment, a containerized network element operating systemprograms updates to the hardware tables in the data plane using a devicedriver. Because the host operating system does not know, a priori, whichnetwork element operating system will eventually be running on thenetwork element, the network element may not be able to program thehardware tables of the network element. Instead, the network elementoperating system can dynamically install a device driver in the kernelof the host operating system. In this embodiment, the network elementoperating system probes the hardware of the data plane (e.g., a type ofASIC used as a hardware forwarding engine for the data plane) andinstalls the device driver in the kernel of the host operating system.With the installed device driver, the containerized network elementoperating system can program updates to the hardware tables in the dataplane.

In a further embodiment, multiple containers of a host operating systemfor a device can be used to simulate a network topology of networkelements. In this embodiment, the host operating system of a device caninstantiate multiple containers, were each of the containers is used toexecute a set of processes to simulate one or more network elements fora given configuration and network topology. Alternatively, each of thesemultiple containers can be instantiated by another container or someother mechanism.

FIG. 1 is a block diagram of one embodiment of a network element 100that includes a control plane 104 and a data plane 102, with the dataplane 102 communicating data to the control plane 104. In oneembodiment, the data plane 102 receives, processes, and forwards networkdata using various configuration data (e.g. packet forwarding (routing,switching, or another type of packet forwarding), security, quality ofservice (QoS), and other network traffic processing information). Forexample, for each received packet of the network traffic, the data planedetermines a destination address of that packet, looks up the requisiteinformation for that destination in one or more hardware tables 120A-Cstored in the data plane, and forwards the packet out the properoutgoing interface. The data plane 102 includes multiple switches 106A-Cthat can each receive, process, and/or forward network traffic. In oneembodiment, each switch 106A-C includes a hardware forwarding engine112A-C and ports 110A-C, respectively. In one embodiment, the networkelement 100 can be a switch, router, hub, bridge, gateway, etc., or anytype of device that can communicate data packets with a network. In oneembodiment, the network element 100 can be a virtual machine.

In one embodiment, the control plane 104 includes central processingunit (CPU) 108. As discussed herein, CPU 108 is interchangeably referredto as a control plane processor of network element 100. The CPU 108 isused to process information for the control plane 104 and writeconfiguration data for hardware forwarding engines 112A-C in theswitches 106A-C. The information processed by CPU 108 includes, forexample, control plane data corresponding to a plurality of differentclasses of control plane traffic, such as routing protocol messages,routing table messages, routing decisions messages, route updatemessages, unresolved traffic messages, L2 protocol messages, linkaggregation control protocol messages, link layer state updates messages(e.g., spanning tree messages), link state update messages (e.g., linkaggregation control protocol messages for a link aggregation group,bidirectional forwarding detection messages, etc.), exception packetsthat cannot be dealt with in hardware (e.g., router alerts, transmissiontime interval messages, maximum transmission size exceeded messages,etc.), program messages (e.g., packets from a controller instructing theprogramming of a network element), messages for routing table misses,time control messages (e.g., precision time protocol messages), messagesfor packets marked as being of interest for snooping (e.g., accesscontrol list logging and port mirroring messages), messages used tocollect traffic diagnostics, address resolution messages (ARP) requestsand replies, neighbor solicitation requests and replies, generalcommunication to the control plane of the networking device, etc. CPU108 processes the control plane network data to perform controlmanagement updates and/or respond with control message responses (e.g.,routing decisions, protocol updates, traffic resolutions, etc.).

In one embodiment, the control plane 104 further includes memory 114that includes operating system 118 that is executing various processes.In this embodiment, the processes 116 are processes that execute thefunctionality of the control plane 104. In one embodiment, there can beprocesses 116 for quality of service, access control lists management(or other types of security), policy service, fan agent, light emittingdiode agent, temperature sensor agent, database service, managementservice(s), processes to support networking protocols (e.g. STP, routingprotocols (e.g. such as RIP, BGP, OSPF, IS-IS, IGRP, EIGRP, PIM, DVMRP,and any/or other type or unicast or multicast routing protocol), MPLS,and/or other types of networking protocols), network flow managementapplications (e.g., openflow, directflow), process manager, and/or othertypes of processes for other types of functionality of the networkelement 100.

In one embodiment, the data plane 102 receives, processes, and forwardsnetwork data, including control plane network data, using variousconfiguration data (e.g., forwarding, security, quality of service(QoS), and other network traffic processing information). The data plane102 includes multiple switches 106A-C that can each receive, process,and/or forward network traffic. Each of the switches 106A-C includesmultiple ports 116A-C that are used to receive and transmit networkdata.

In one embodiment, for each received unit of network data (e.g., apacket), the data plane 102 determines a destination address for thenetwork data, looks up the requisite information for that destination inone or more tables stored in the data plane, and forwards the data outthe proper outgoing interface, for example, one of the interface devices106A-C. In one embodiment, each switch 106A-C includes one or morehardware forwarding engines (HWFE(s)) 112A-C and ports 116A-C,respectively. Each hardware forwarding engine 112A-C forwards data forthe network element 100, such as performing routing, switching, or othertypes of network forwarding or processing.

In one embodiment, for each received unit of control plane data, thedata plane 102 forwards the control plane network data to the CPU 108(e.g., the control plane processor). In one embodiment, the controlplane 104 gathers configuration data for the hardware forwarding engines112A-C in control plane network data messages from different sources(e.g., locally stored configuration data, via a command line interface,or other management channel (e.g., SNMP (Simple Network ManagementProtocol), Simple Object Access Protocol (SOAP), Representational StateTransfer type Application Programming Interface (RESTful API), HypertextTransfer Protocol (HTTP), HTTP over Secure Sockets layer (HTTPs),Network Configuration Protocol (NetConf), Secure Shell (SSH), and/oranother management protocol) and pushes this configuration data to thehardware forwarding engines 112A-C.

The data plane 102 further includes hardware tables 120A-C that are oneor more tables used to configure the processing of the incoming handledby the hardware forwarding engines 112A-C. In one embodiment, thehardware tables 120A-C are used to store configuration data, monitoringdata, reporting data, statistics, and any other data the hardwareforwarding engine uses or collects. In one embodiment, the hardwaretables 120A-C can include a routing table, MAC table, ACL, and othertables. For example and in one embodiment, the routing table storesrouting table entries which may be produced by any of the runningprotocols known in the art such as RIP, BGP, OSPF, IS-IS, IGRP, EIGRP,PIM, DVMRP, and any/or other type or unicast or multicast routingprotocol known in the art. The routing table entries may be used toforward packets that are encoded with one of the address families knownin the art such as IPv4 unicast, IPv6 unicast, IPv4 multicast, or IPv6multicast. The MAC table is a table of MAC addresses known to thehardware forwarding engine and Virtual Local Area Network (VLAN)s andports that these MAC address are associated with. In one embodiment, theMAC table is used for layer 2 forwarding. In another embodiment, the MACtable records a MAC address and VLAN and an associated tunnel interface.This tunnel may be a Virtual eXtensible LAN (VXLAN) tunnel, a GenericRouting Encapsulation (GRE) tunnel, an L2TP tunnel, an Internet Protocol(IP)-in-IP tunnel, a Multiprotocol Label Switching (MPLS) label, or oneof any number of tunneling formats known in the art. In one embodiment,the ACL consists of an ordered series of rules, where each rule has amatch criterion and action. In this embodiment, the ACL 206 is appliedto the network data against these rules in order, and taking the actionof the first rules that matches. In one embodiment, the other tables areset of one or more tables that is used to store statistics, monitoringdata, other configuration data, stored events, management data, and/orother data the hardware forwarding engine uses or collects.

Containerized Network Element Operating System

FIG. 2 is a block diagram of one embodiment of a network element 200with a network element operating system executing within a container210A. In FIG. 2, network element 200 includes a control plane 202coupled to a data plane 204. In one embodiment, the data plane 204includes one or more hardware forwarding engines 216 and one or morehardware tables 218. In one embodiment, the hardware forwarding engines216 are the hardware forwarding engines as described in FIG. 1 above.Furthermore, in this embodiment, the hardware tables 218 are thehardware tables as described in FIG. 1 above. In one embodiment, thedata plane 204 processes the incoming data 220 using the hardwareforwarding engines 216 according to the data in the hardware tables 218.In this embodiment, the hardware forwarding engines 216 decide whetherto forward the data to a next hop, forward the data to the control plane202, and/or further process the data. If the hardware forwarding engine216 determines that the incoming data is control plane data, thehardware forwarding engine 216 forwards this control plane data 222 tothe network element operating system process(es) 212A, where the networkelement operating system process(es) 212A processes this control planedata 222. If the incoming data 220 is data plane data, the hardwareforwarding engines 216 can determine the next hop for this data and/orperform further processing of this data (e.g., apply a quality service,perform traffic shaping, process with relevant ACLs, and/or otherprocessing).

The control plane 202, in one embodiment, includes a CPU 206 and memory208. In this embodiment, the memory 208 includes a host operating system214 and two containers 210A-B. Each of the containers 210A-B can each beinstantiated by the host operating system 214, another container, orsome other mechanism. In one embodiment, a container 210A-B is anamespace instance of an operating system level virtualization. In thisembodiment, the container 210A-B is used for deploying and runningprocesses without launching an entire virtual machine for each of theseprocesses. Instead, these multiple isolated containers are run on asingle control host operating system 214 and access the kernel of thehost operating system 214. Because these containers 210A-B share thesame host operating system kernel as the network element 200, thecontainers can be more efficient than virtual machines, which requireseparate operating system instances, including a kernel instantiationfor each of the virtual machines. In addition, each virtual machine isrequired to be run on a hypervisor. In one embodiment, the containers210A-B do not require a separate kernel or a hypervisor to execute aprocess within this container. The containers 210A-B include thecomponents used to run the desired software, such as files,environmental variables, data, metadata, and/or libraries. In oneembodiment, the containers 210A-B are compatible with a DOCKER type ofcontainers. In this embodiment, the host operating system 214 canadditionally manage hardware of the network element that is differentfrom the networking functions of the network element 200. For exampleand in one embodiment, the network element 200 may have other hardwareand/or other functions (e.g., storage systems, security hardware,different hypervisors, and/or other types of functions) that are beingmanaged by the host operating system. In one embodiment, thisarchitecture decouples the network hardware and services with thenetwork element operating system from the host operating system.

As described above, the memory 208 includes the network elementoperating system container 210A and further can include a third-partycontainer 210B. In one embodiment, the network element operating systemcontainer 210A is an isolated namespace used to execute the networkelement operating system process(es) 212A, where the network elementoperating system process(es) 212A can be used to manage the data storedin the hardware tables 218 and/or other management functionalities ofthe network element 200. By executing the network element operatingsystem process(es) 212A in the network element operating systemcontainer 210A, a network element operating system can be run on anetwork element that is from a different manufacturer than the one thatproduced the network element operating system. For example and in oneembodiment, a network element operating system from one manufacturer canbe run on a white box network element that has a different manufacturer.Thus, instead of a network element operating system acting as the hostoperating system for the network element 200, the network elementoperating system acts as an application executing on the network element200.

In one embodiment, the network element operating system process(es) 212Areceive the control plane data 222 from the hardware forwarding engines216 via the host operating system 214. The network element operatingsystem process(es) 212A receive the control plane data 222, processesthe control plane data 222, and programs the processed control planedata 224 to the hardware tables 218 of the data plane 204. In thisembodiment, by programming updates to the hardware tables 218, thenetwork element operating system process(es) 212A controls how the dataplane 204 of the network element performs.

As with a network element operating system executing natively on anetwork element 200, the network element operating system can beremotely managed. In one embodiment, the network element 200 includes amanagement interface (not illustrated) that is used by remote client tomanage and/or configure the operations of the network element 200. Inthis embodiment, because the network operating system is executing as acontainer on top of a host operating system 214, a mechanism is neededto allow the network element operating system process(es) to communicatemanagement data with the remote client. In one embodiment, this can bedone either by the network element operating system being private, orisolated, and the host operating system 214 visible to the remoteclient, the network element operating system visible to the remoteclient and the host operating system 214 being isolated, or both thenetwork element operating system and the host operating system 214 eachvisible to the remote client.

In one embodiment, the network operating system process(es) 212A areprivate to a remote client, because a remote client cannot access theseprocess(es) 212A via a management interface. In this embodiment, thenetwork 200 includes a private network between the host operating system214 and the network element operating system process(es) 212A that isused to communicate management data with the network element operatingsystem process(es) 212A. Thus, a remote client wishing to communicatewith the management processes (e.g., via a command line interface withthe network element operating system process(es) 212A) communicates withthe management interface of the network element 200. In this embodiment,management data that is received on the network element managementinterface is forwarded over the private network to the network elementoperating system process(es) 212A, where the network element operatingsystem process(es) 212A process this management data and returns theoutput (if appropriate). (In one embodiment, the host operating system214 implements a Network Address Translation (NAT) service and accessesspecific processes in the network element operating service process(es)212A by mapping ports attached to some or all of the network elementoperating service process(es) 212A (e.g. a port for SSH). In thisembodiment, a network administrator can access the network elementoperating service process(es) 212A using SSH, SNMP, or another type ofmanagement mechanism.

In an alternate embodiment, the host operating system 214 is isolatedand the network element operating system process(es) 212A is availableto a remote client and further manages the management interface. In thisembodiment, the network element 200 bridges the physical managementinterface to a virtual interface inside the network element operatingsystem container 210A. Thus, the network element operating systemprocess(es) 212A initially receives the management data (e.g. CLIcommands or other management requests) and forwards any results fromthis management data back to the remote client by the internal bridge ofthe network element and the management interface. In this embodiment,the host operating system 214 is available via console (e.g., a physicalconsole interface or console server).

In a further embodiment, both the host operating system 214 and thenetwork element operating system process(es) 212A are visible to aremote client. In this embodiment, each of the host operating system 214and the network element operating system process(es) 212A would havedifferent interfaces assigned to them that are accessible by the remoteclient. For example and in one embodiment, a MAC VLAN interface is usedto expose to reachable endpoints sharing the same physical port for thehost operating system 214 and the network element operating systemprocess(es) 212A.

In one embodiment, there can be other containers for other processesthat are instantiated, with executing processes, that may or may not befrom the same vendor as the network element operating system or the hostoperating system 214. In one embodiment, a third-party process 212B canbe executing within a third-party container 210B concurrently with thenetwork element operating system process(es) 212A. In this embodiment,the third-party container 210B can be used to implement other functionsof the network, such as other types of management for the networkelement 200 that is not currently be handled by the network elementoperating system process(es) 212A. Thus, the network administrator couldchoose a white box network element 200 from one vendor, installing ahost operating system 214 from a second vendor, have this white boxnetwork element 200 run a network element operating system from a thirdvendor in a container 210A that are executing the network elementoperating system process(es) 212A, and have a third-party applicationfrom a fourth vendor executing in different container 210B.

FIGS. 3A-B are flow diagrams of embodiments of processes to instantiatea container for a network element operating system and to processcontrol plane data using network operating system process(es) withinthis container. In FIG. 3A, process 300 begins by receiving a networkelement operating system container image at block 302. In one embodimentthe network element operating system container image is an image thatcan be used to instantiate one or more processes that are used by anetwork element operating system to run and manage a network element. Inparticular, the network element operating system can be used to programand maintain a data plane of the network element, so that the networkelement can process and forward data that the network element receives.At block 304, process 300 instantiates a container that will be used forthe network element operating system processes. In one embodiment, theinstantiated container provides an operating system level virtualizationfor these processes. Process 300 instantiates the network elementoperating system process(es) at block 306. In one embodiment, thesenetwork element operating system process(es) are used by the networkelement operating system to run and manage this network element.

In one embodiment, with the network element operating system process(es)executing in the network element operating system container, thisnetwork element operating system can be used to manage and update a dataplane of the network element. FIG. 3B is a flow diagram of oneembodiment of a process 350 to process control plane data and programupdates to the hardware tables of a data plane of a network element. InFIG. 3B, process 350 begins by receiving control plane data from ahardware forwarding engine of the data plane of the network element atblock 352. In one embodiment, control plane data is data that is to beprocessed by the control plane (e.g., routing protocol updates, addressupdates, configuration data, statistics, and/or other data processed bythe control plane) as described above. At block 354, process 350processes the control plane data and determines a hardware table update.In one embodiment, the hardware table update could be for a routingtable, MAC address table, ACL list, and/or another update to anothertable. Process 350 programs the hardware table(s) using the hardwaretable update at block 356. In one embodiment, by the network elementoperating system process(es) programming the hardware table using thehardware table updates, the network element operating system process(es)can manage and control the data plane of the network element.

FIG. 4 is block diagram of one embodiment of a network element 400 witha network element operating system executing within multiple containers410A-N. In FIG. 4, network element 400 includes a control plane 402coupled to a data plane 404. In one embodiment, the data plane 404includes one or more hardware forwarding engines 416 and one or morehardware tables 418. In one embodiment, the hardware forwarding engines416 are the hardware forwarding engines as described in FIG. 1 above.Furthermore, in this embodiment, the hardware tables 418 are thehardware tables as described in FIG. 1 above. In one embodiment, thedata plane 404 processes the incoming data 420 using the hardwareforwarding engines 416 according to the data in the hardware tables 418.In this embodiment, the hardware forwarding engines 416 decide whetherto forward the data to a next hop, forward the data to the control plane402, and/or further process the data. If the hardware forwarding engine416 determines that the incoming data is control plane data, thehardware forwarding engine 416 forwards this control plane data 422 tothe network element operating system process(es) 412A, where the networkelement operating system process(es) 412A processes this control planedata 422. If the incoming data 420 is data plane data, the hardwareforwarding engines 416 can determine the next hop for this data and/orperform further processing of this data (e.g., apply a quality service,perform traffic shaping, process with relevant ACLs, and/or otherprocessing).

The control plane 402, in one embodiment, includes a CPU 406 and memory408. In this embodiment, the memory 408 includes a host operating system414 and containers 410A-N that are each instantiated by the hostoperating system 414, another container, or some other mechanism. In oneembodiment, a container 410A-N is a namespace instance of an operatingsystem level virtualization. In this embodiment, the container 410A-N isused for deploying and running processes without launching an entirevirtual machine for each of these processes. Instead, these multipleisolated containers are run on a single control host operating system414 and access the kernel of the host operating system 414. Becausethese containers 410A-N share the same host operating system kernel asthe network element 400, the containers can be more efficient thanvirtual machines, which require separate operating system instances. Thecontainers 410A-N include the components used to run the desiredsoftware, such as files, environmental variables, data, metadata, and/orlibraries. In one embodiment, the containers 410A-B are compatible withthe DOCKER types of containers.

As described above, the memory 408 includes the network elementoperating system containers 410A-N and further can include a third-partycontainer 426 with a third-party process 428 that executes within thethird-party container 426. In one embodiment, the functionalities of thenetwork element operating system are split into multiple containers410A-N, where there one or more network element operating systemprocess(es) 412A-N per container 410A-N. In this embodiment, each of thenetwork element operating system containers 410A-N is an isolatednamespace used to execute the network element operating systemprocess(es) 412A-N, where the network element operating systemprocess(es) 412A-N can be used to manage the data stored in the hardwaretables 418 and/or other management functionalities of the networkelement 400. By executing the network element operating systemprocess(es) 412A-N in the network element operating system containers410A-N, a network element operating system collectively can be run on anetwork element that is from a different vendor than the one thatproduced the network element operating system. For example and in oneembodiment, a network element operating system from one manufacturer canbe run on a white box network element that has a different manufacturer.Thus, instead of a network element operating system acting as the hostoperating system for the network element 400, the network elementoperating system acts as an application executing on the network element400.

In FIG. 4, with the functionality of the network element operatingsystem split across multiple network element operating systemprocess(es), each of the processes 412A-N in the different containers410A-N can act like a separate application and can be upgradedseparately. In one embodiment, one of the containers 410A-N is used toexecute a database process that receives and stores state data of thenetwork element operating system processes 412A-N. in this embodiment,the database process is used to store various state data that is furtherused by the control plane to manage the network element 400.

In a further embodiment, each of the network element operating systemprocesses 412A-N can share libraries from one or more layers for imagecompatibility and reduced image size. In this embodiment, theselibraries (or other types of binaries) are shared even when the networkoperating system is partitioned among multiple containers 410A-N. Forexample and in one embodiment, a routing container will have layers,such as a routing layer where a routing process executes, and a basiclayer that is utilized by the routing layer. In a further example,hardware container will have the layers of the hardware agents and thebasic layer. In these examples, the binary for the basic layer used bythe routing and the hardware containers is the same binary, which leadsto a reduced image size for these network element operating systemprocesses it is also used for the image compose ability for theseprocesses.

In a further embodiment, these multiple container 410A-N instances areused to manage different aspects of the network element. For example andin one embodiment, there can be processes 412A-N for the quality ofservice, access control lists management (or other types of security),policy service, fan agent, light emitting diode agent, temperaturesensor agent, database service, management service(s), processes tosupport networking protocols (e.g. STP, routing protocols (e.g. such asRIP, BGP, OSPF, IS-IS, IGRP, EIGRP, PIM, DVMRP, and any/or other type orunicast or multicast routing protocol), MPLS, and/or other types ofnetworking protocols), network flow management applications (e.g.,openflow, directflow), process manager, and/or other types of processesfor other types of functionality of the network element 400.

In one embodiment, each of the network element operating systemprocesses 412A-N can open one or more sockets (e.g., a RAW socket,another type of socket, or communication port) on the ports in thekernel of the host operating system and listen for the type of data thatprocess is interested in. In another embodiment, a platform driver canbe installed into the kernel of the host operating system that offers adifferent interface to the associated process. In a further embodiment,if a service of one of the network element operating service process(es)212A-N is over Transport Control Protocol (TCP), this service willterminate inside the container.

FIGS. 5A-B are flow diagrams of embodiments of processes to instantiatemultiple containers for a network element operating system and toprocess control plane data using network operating system process(es)within these containers. In FIG. 5A, process 500 receives the networkelement operating system container images at block 502. In oneembodiment, the network element operating system container images areimages that can be used to instantiate one or more processes thatcollectively are used by a network element operating system to run andmanage a network element. In particular, the network element operatingsystem can be used to program and maintain a data plane of the networkelement, so that the network element can process and forward data thatthe network element receives. In this embodiment, these images caninclude libraries that are used by multiple processes. At block 504,process 500 selects which images to instantiate. In this embodiment,process 500 determines which images to instantiate based on theconfiguration for the network element. The network element may not beconfigured to run each possible service that is available within thenetwork element operating system. Thus, process 500 selects the imagesfurther configured services, agents, and/or other processes that thenetwork element is configured to do. At block 506, process 500instantiates the selected containers for these configured services,agents, and/or processes. In one embodiment, the configuration of thenetwork element includes which of these services, agents, and/orprocesses are to be executed and which of the instantiated containers.Process 500 instantiates processes for the configure services, agents,and/or processes in the appropriate selected containers at block 508.

In one embodiment, with the network element operating system process(es)executing in the network element operating system containers, thisnetwork element operating system can be used to manage and update a dataplane of the network element. FIG. 5B is a flow diagram of oneembodiment of a process 550 to process control plane data and programupdates to the hardware tables of a data plane of a network element. InFIG. 5B, process 550 begins by receiving control plane data from ahardware forwarding engine of the data plane of the network element atblock 552. At block 554, process 550 processes the control plane dataand determines a hardware table update. Process 550 programs thehardware table(s) using the hardware table update at block 556. In oneembodiment, by the network element operating system process(es)programming the hardware table using the hardware table updates, thenetwork element operating system process(es) can manage and control thedata plane of the network element.

Hitless Upgrades of a Containerized Network Operating System

Periodically, the software image of a network element is updated. Forexample, the network element can be upgraded to a new software image toadd new features and/or fix problems with the current software image.Alternatively, the network image can be downgraded to a previous versionof the software image. To change the software image, the network elementloads the new software image, marks this new software image as thecurrent software image, and reboots the network element. As the networkelement boots up, the new software image is loaded and programs thecontrol and data planes of the network element.

A problem with this way of upgrading or downgrading the software imageis that, by rebooting the network element, the network element isinoperable for a relatively long time. For example, rebooting thenetwork element can cause the network element to be off-line for upwardsof five minutes. During this off-line time, the data processingfunctions of the network element are inoperable. This can lead to aninterruption of data networking services that rely on the networkelement. If there can be a hitless upgrade for a network element, wherethe network element is not rebooted, then the data processing of thedata plane is not interrupted.

FIG. 6 is a block diagram of one embodiment of a network element 600performing a hitless upgrade for a network element operating systemexecuting within a container 610A of a host operating system 614. InFIG. 6, network element 600 includes a control plane 602 coupled to adata plane 604. In one embodiment, the data plane 604 includes one ormore hardware forwarding engines 616 and one or more hardware tables618. In one embodiment, the hardware forwarding engines 616 are thehardware forwarding engines as described in FIG. 1 above. Furthermore,in this embodiment, the hardware tables 618 are the hardware tables asdescribed in FIG. 1 above. In one embodiment, the data plane 604processes the incoming data 620 using the hardware forwarding engines616 according to the data in the hardware tables 618. In thisembodiment, the hardware forwarding engines 616 decide whether toforward the data to a next hop, forward the data to the control plane602, and/or further process the data. If the hardware forwarding engine616 determines that the incoming data is control plane data, thehardware forwarding engine 616 forwards this control plane data 622 tothe network element operating system process(es) 612A, where the networkelement operating system process(es) 612A processes this control planedata 622. If the incoming data 620 is data plane data, the hardwareforwarding engines 616 can determine the next hop for this data and/orperform further processing of this data (e.g., apply a quality service,perform traffic shaping, process with relevant ACLs, and/or otherprocessing).

The control plane 602, in one embodiment, includes a CPU 606 and memory608. In this embodiment, the memory 608 includes a host operating system614 and two containers 610A-B that are each instantiated by the hostoperating system 614, another container, or some other mechanism. In oneembodiment, a container 610A-B is a namespace instance of an operatingsystem level virtualization. In this embodiment, the container 610A-B isused for deploying and running processes without launching an entirevirtual machine for each of these processes. Instead, these multipleisolated containers are run on a single control host operating system614 and access the kernel of the host operating system 614. Becausethese containers 610A-B share the same host operating system kernel asthe network element 600, the containers can be more efficient thanvirtual machines, which require separate operating system instances. Thecontainers 610A-B include the components used to run the desiredsoftware, such as files, environmental variables, data, metadata, and/orlibraries. In one embodiment, the containers 610A-B are compatible withthe DOCKER types of containers. As described above, the memory 608includes the network element operating system container 610A and furthercan include a third-party container 610B with a third-party process 612Bthat executes within the third-party container 610B. In one embodiment,the network element operating system container 610A is an isolatednamespace used to execute the network element operating systemprocess(es) 612A, where the network element operating system process(es)612A can be used to manage the data stored in the hardware tables 618and/or other management functionalities of the network element 600. Byexecuting the network element operating system process(es) 612A in thenetwork element operating system container 610A, a network elementoperating system can be run on a network element that is from adifferent vendor than the one that produced the network elementoperating system. For example and in one embodiment, a network elementoperating system from one vendor can be run on a white box networkelement that has a different vendor. Thus, instead of a network elementoperating system acting as the host operating system for the networkelement 600, the network element operating system acts as an applicationexecuting on the network element 600.

In one embodiment, with a containerized network element operating systemin one or more containers, a hitless upgrade can be performed byinstantiating a new container 626 for the network element operatingsystem, instantiate the one or more processes 628 for this container,synchronizing a state between the current network element operatingsystem processes 628 and the new network element operating systemprocesses 612A in the new container, and switching over control from thecurrent network element operating system processes 612A to the networkelement operating system processes 628 in the new container 626. In thisembodiment, the switching over to the new container 626 can include thenew network element operating system processes 628 taking over controlof a management interface or management network used by the networkelement operating system processes 612A and stopping the current networkelement operating system processes 612A and deleting the current networkelement operating system containers 610A. In one embodiment, the stateof the network element operating system being upgraded can be storedoutside the affected container in other containers, such as a databasecontainer, or in a shared memory of an underlying system. In a furtherembodiment because the container itself is stateless, the new containerand associated process(es) can instantly take over the processing of therelevant control plane data without a hit to the data processing ofincoming data.

FIG. 7 is a flow diagrams of one embodiment of a process 700 to performa hitless upgrade of a network element operating system executing withina container of a host operating system. In FIG. 7, process 700 begins byreceiving a new image for a hitless upgrade of the network elementoperating system, where a current image of the network element operatingsystem is running in a first container. In one embodiment, the new imagecan be a software upgrade to the network element operating system,software downgrade to the network element operating system, or just areinstallation of the same version of the network element operatingsystem that is currently running on the network element. At block 704,process 700 instantiates a new container for the new network elementoperating system image. Process 700 starts the one or more networkelement operating system processes in the new container at block 706. Atthis point, the network element has two containers executing the networkelement operating system, where the network element operating system inthe current container is the active network element operating system andthe network element operating system executing in the new container willbe the new version of the network element operating system once theswitchover is made.

At block 708, process 700 synchronizes state data with the currentnetwork element operating system processes in the current container. Inone embodiment, by synchronizing the state data with the current networkelement operating system processes, the new network element operatingsystem processes will have the same or similar state as in the currentnetwork element operating system processes. While in one embodiment, thenew network element operating system process(es) synchronize the statewith the current network element operating system process(es), inalternate embodiments, the new network element operating systemprocess(es) can rebuild a state from data stored in the network element.In one embodiment, process 700 sets the new container, and thecorresponding processes, as the current network element operatingsystem. In one embodiment, process 700 sets the new container, and thecorresponding processes, as the current network element operating systemby having these processes take control of the management mechanism forthe network element operating system so that any new control plane dataflows to the now current network element operating system processes inthe newly instantiated container. At block 710, process 700 deletes thepreviously current container, which stops the previously current networkelement operating system processes and frees the memory used by theseprocesses, as well as releasing any other resources used by theprocesses. At this point, the network element is executing the updatednetwork element operating system without interrupting the processing ofthe data plane, because the network element did not need to be rebootedand the data plane processing did not need to be otherwise interrupted.

FIG. 8 is a block diagram of one embodiment of a network element 800performing a hitless upgrade for a network element operating systemfunction executing within one of multiple containers of a host operatingsystem. FIG. 8 is block diagram of one embodiment of a network element800 with a network element operating system executing within multiplecontainers 810A-N of a host operating system 814. In FIG. 8, networkelement 800 includes a control plane 802 coupled to a data plane 804. Inone embodiment, the data plane 804 includes one or more hardwareforwarding engines 816 and one or more hardware tables 818. In oneembodiment, the hardware forwarding engines 816 are the hardwareforwarding engines as described in FIG. 1 above. Furthermore, in thisembodiment, the hardware tables 818 are the hardware tables as describedin FIG. 1 above. In one embodiment, the data plane 804 processes theincoming data 820 using the hardware forwarding engines 816 according tothe data in the hardware tables 818. In this embodiment, the hardwareforwarding engines 816 decide whether to forward the data to a next hop,forward the data to the control plane 802, and/or further process thedata. If the hardware forwarding engine 816 determines that the incomingdata is control plane data, the hardware forwarding engine 816 forwardsthis control plane data 822 to the network element operating systemprocess(es) 812A, where the network element operating system process(es)812A processes this control plane data 822. If the incoming data 820 isdata plane data, the hardware forwarding engines 816 can determine thenext hop for this data and/or perform further processing of this data(e.g., apply a quality service, perform traffic shaping, process withrelevant ACLs, and/or other processing).

The control plane 802, in one embodiment, includes a CPU 806 and memory808. In this embodiment, the memory 808 includes a host operating system814 and containers 810A-N that each can be instantiated by the hostoperating system 814, another container, or come other mechanism. In oneembodiment, a container 810A-N is a namespace instance of an operatingsystem level virtualization. In this embodiment, the container 810A-N isused for deploying and running processes without launching an entirevirtual machine for each of these processes. Instead, these multipleisolated containers are run on a single control host operating system814 and access the kernel of the host operating system 814. Becausethese containers 810A-N share the same host operating system kernel asthe network element 800, the containers can be more efficient thanvirtual machines, which require separate operating system instances. Thecontainers 810A-N include the components used to run the desiredsoftware, such as files, environmental variables, data, metadata, and/orlibraries. In one embodiment, the containers 810A-N are compatible withthe DOCKER types of containers.

As described above, the memory 808 includes the network elementoperating system containers 810A-N and further can include a third-partycontainer 828 with a third-party process 830 that executes within thethird-party container 828. In one embodiment, the functionalities of thenetwork element operating system are split into multiple containers810A-N, where there one or more network element operating systemprocess(es) 812A-N per container 810A-N. In this embodiment, each of thenetwork element operating system containers 810A-N is an isolatednamespace used to execute the network element operating systemprocess(es) 812A-N, where the network element operating systemprocess(es) 812A-N can be used to manage the data stored in the hardwaretables 818 and/or other management functionalities of the networkelement 800. By executing the network element operating systemprocess(es) 812A in the network element operating system containers810A-N, a network element operating system collectively can be run on anetwork element that is from a different vendor than the one thatproduced the network element operating system. For example and in oneembodiment, a network element operating system from one vendor can berun on a white box network element that has a different vendor. Thus,instead of a network element operating system acting as the hostoperating system for the network element 800, the network elementoperating system acts as an application executing on the network element800.

In FIG. 8, with the functionality of the network element operatingsystem split across multiple network element operating systemprocess(es), each of the processes 812A-N in the different containers810A-N can act like a separate application that can be upgradedseparately. In one embodiment, one of the containers 810A-N is used toexecute a database process that receives and stores state data of thenetwork element operating system processes 812A-N. In this embodiment,the database process is used to store various state data that is furtherused by the control plane to manage the network element 800.

In a further embodiment, each of the network element operating systemprocesses 812A-N can share libraries from one or more layers for imagecompatibility and reduced image size. In this embodiment, theselibraries (Inventor Q: are there other types of binaries? Not sure ifthis has been answered before) are shared even when the networkoperating system is partitioned among multiple containers 810A-N. Forexample and in one embodiment, a routing container will have layers,such as a routing layer where a routing process executes, and a basiclayer that is utilized by the routing layer. In a further example,hardware container will have the layers of the hardware agents and thebasic layer. In these examples, the binary for the basic layer used bythe routing and the hardware containers is the same binary, which leadsto a reduced image size for these network element operating systemprocesses it is also used for the image compose ability for theseprocesses.

In a further embodiment, these multiple container 810A-N instances areused to manage different aspects of the network element. For example andin one embodiment, there can be processes 812A-N for the quality ofservice, access control lists management (or other types of security),policy service, fan agent, light emitting diode agent, temperaturesensor agent, database service, management service(s), processes tosupport networking protocols (e.g. STP, routing protocols (e.g. such asRIP, BGP, OSPF, IS-IS, IGRP, EIGRP, PIM, DVMRP, and any/or other type orunicast or multicast routing protocol), MPLS, and/or other types ofnetworking protocols), network flow management applications (e.g.,openflow, directflow), process manager, and/or other types of processesfor other types of functionality of the network element 800.

With the network element operating system functionality partitionedacross different network element operating system containers 810A-N,upgrading of the network element operating system occurs by upgradingthe image used by the specific network element operating systemprocesses 812A-N into a new container 826 with a new executing process832. In this embodiment, if the network element operating system upgradeaffects a subset of the processes 812A-N, then the affected processesare the ones that get upgraded, without affecting the other processes.For example and of one embodiment, if process 812A is a BGP process ofthe network element operating system in this process has a critical bugfix that is available, a hitless upgrade can be accomplished byinstantiating a container for a BGP image with this critical bug fix,starting a new BGP process within the new container, synchronizing astate between the current BGP process and the new BGP process, andswitching control for the BGP processing of the network elementoperating system to the newly started BGP process. In one embodiment,because BGP is over TCP, when the container comes up, the new BGPprocess will establish a TCP session and re-synchronize with the remotehost(s). In response to the re-synchronization process, the new BGPprocess takes over processing of any new BGP updates. In thisembodiment, the other processes of the network element operating systemare unaffected and can continue to perform their functionality and/orprocess relevant control plane data. In one embodiment, the state of theprocess being upgraded can be stored outside the affected container inother containers, such as a database container, or in a shared memory ofan underlying system. In a further embodiment because the containeritself is stateless, the new container and associated process(es) caninstantly take over the processing of the relevant control plane datawithout a hit to the data plane processing.

FIG. 9 is a flow diagrams of one embodiment of a process to perform ahitless upgrade for a network element operating system functionexecuting within one of multiple containers of a host operating system.In FIG. 9, process 900 begins by receiving a new image for a hitlessupgrade of the network element operating system function, where acurrent image of the network element operating system function isrunning in a first container. In one embodiment, the new image can be asoftware upgrade to the network element operating system function,software downgrade to the network element operating system function, orjust a reinstallation of the same version of the network elementoperating system function that is currently running on the networkelement. In this embodiment, the network element operating systemfunction is currently executing as one or more processes in the firstcontainer. For example and in one embodiment, the image could be for anew version of the BGP service.

At block 904, process 900 instantiates a new container for the newnetwork element operating system image of that function. Process 900starts the one or more network element operating system processes forthat function in the new container at block 906. At this point, thenetwork element has two containers executing the network elementoperating system function, where the network element operating system inthe current container is the active network element operating system forthat function and the network element operating system executing in thenew container will be the new version of the network element operatingsystem function once the switchover is made. For example and in oneembodiment, the network element, at this point includes two BGPprocesses executing in two different containers, where the current BGPprocesses in the active process and processing BGP updates and the newBGP process is waiting to eventually take over.

At block 908, process 900 synchronizes state data with the currentnetwork element operating system processes in the current container forthat function. In one embodiment, by synchronizing the state data withthe current network element operating system processes for thatfunction, the new network element operating system processes will havethe same state as in the current network element operating systemprocesses for this function. Process 900 sets the new container, and thecorresponding processes, as the current network element operating systemfor that function. In one embodiment process 900 sets the new container,and the corresponding processes, as the current network elementoperating system by synchronization data with remote host (e.g., BGP),taking over a socket for the process, taking over control of a platformdriver that handles delivering messages to different containers, and/orsome other action. In one embodiment, a new container can become up andrunning is less that one second. For example and in one embodiment,process 900 has the new BGP process take over responsibility forprocessing of any future BGP updates from the current BGP process.

At block 910, process 900 deletes the previously current container,which stops the previously current network element operating systemfunction processes and frees the memory used by these processes. At thispoint, the network element is executing the updated network elementoperating system function without interrupting the processing of thedata plane, because the network element did not need to be rebooted andthe data plane processing did not need to be otherwise interrupted. Forexample and in one embodiment, is there are other routing protocolprocesses currently executing different containers during the BGPprocess upgrade, these other routing processes are unaffected becausethe BGP process is the only process being upgraded.

Dynamically Installing a Device Driver in a Network Element

FIG. 10 is a block diagram of one embodiment of a network element 1000that performs a device driver installation into a kernel of a hostoperating system 1014 by a network operating system executing in acontainer. FIG. 10 is a block diagram of one embodiment of a networkelement 1000 performing a hitless upgrade for a network elementoperating system executing within a container 1010A of a host operatingsystem 1014. In FIG. 10, network element 1000 includes a control plane1002 coupled to a data plane 1004. In one embodiment, the data plane1004 includes one or more hardware forwarding engines 1016 and one ormore hardware tables 1018. In one embodiment, the hardware forwardingengines 1016 are the hardware forwarding engines as described in FIG. 1above. Furthermore, in this embodiment, the hardware tables 1018 are thehardware tables as described in FIG. 1 above. In one embodiment, thedata plane 1004 processes the incoming data 1020 using the hardwareforwarding engines 1016 according to the data in the hardware tables1018. In this embodiment, the hardware forwarding engines 1016 decidewhether to forward the data to a next hop, forward the data to thecontrol plane 1002, and/or further process the data. If the hardwareforwarding engine 1016 determines that the incoming data is controlplane data, the hardware forwarding engine 1016 forwards this controlplane data 1022 to the network element operating system process(es)1012A, where the network element operating system process(es) 1012Aprocesses this control plane data 1022. If the incoming data 1020 isdata plane data, the hardware forwarding engines 1016 can determine thenext hop for this data and/or perform further processing of this data(e.g., apply a quality service, perform traffic shaping, process withrelevant ACLs, and/or other processing).

The control plane 1002, in one embodiment, includes a CPU 1006 andmemory 1008. In this embodiment, the memory 1008 includes a hostoperating system 1014 and two containers 1010A-B that are eachinstantiated by the host operating system 1014, another container, orsome other mechanism. In one embodiment, a container 1010A-B is anamespace instance of an operating system level virtualization. In thisembodiment, the container 1010A-B is used for deploying and runningprocesses without launching an entire virtual machine for each of theseprocesses. Instead, these multiple isolated containers are run on asingle control host operating system 1014 and access the kernel of thehost operating system 1014. Because these containers 1010A-B share thesame host operating system kernel as the network element 1000, thecontainers can be more efficient than virtual machines, which requireseparate operating system instances. The containers 1010A-B include thecomponents used to run the desired software, such as files,environmental variables, data, metadata, and/or libraries. In oneembodiment, the containers 1010A-B are compatible with the DOCKER typesof containers.

As described above, the memory 1008 includes the network elementoperating system container 1010A and further can include a third-partycontainer 1010B with a third-party process 1012B that executes withinthe third-party container 1012A. In one embodiment, the network elementoperating system container 1010A is an isolated namespace used toexecute the network element operating system process(es) 1012A, wherethe network element operating system process(es) 1012A can be used tomanage the data stored in the hardware tables 1018 and/or othermanagement functionalities of the network element 1000. By executing thenetwork element operating system process(es) 1012A in the networkelement operating system container 1010A, a network element operatingsystem can be run on a network element that is from a different vendorthan the one that produced the network element operating system. Forexample and in one embodiment, a network element operating system fromone vendor can be run on a white box network element that has adifferent vendor. Thus, instead of a network element operating systemacting as the host operating system for the network element 1000, thenetwork element operating system acts as an application executing on thenetwork element 1000.

In one embodiment, the network element operating system process(es)1012A programs the hardware tables 1018 using an installed device driver1026 that is part of kernel 1028 of the host operating system 1014. Inthis embodiment, because the host operating system 1014 does not knowahead of time which type of network element operating system process(es)1012A will be instantiated in the control plane 1002, the networkelement operating system process(es) 1012A installs the device driver1026 in the kernel 1028 based on the type of hardware in the data plane1004. In this embodiment, the network element operating systemprocess(es) 1012A determine the type of hardware in the data plane 1004by invoking a utility of the host operating system to return systeminformation. The network element operating system process(es) 1012Aparses the information from the utility call. In this embodiment, thenetwork element operating system process(es) 1012A can query the devicefamily of ASIC that is part of the data plane 1004. In addition, ahardware agent of the network element operating system process(es) 1012Acan handle minor version differences within an ASIC family (e.g., byinserting a module to make software development kit (SDK) calls for thepurpose of handling different ASICs within an ASIC family). In oneembodiment, the network element operating system process(es) 1012Adetermines which device driver to install based on the type of hardwareforwarding engine 1016 that is part of the data plane 1004. For exampleand in one embodiment, the hardware forwarding engine 1016 can be acertain type of ASIC that is used in the data plane 1004. Based on thetype of ASIC, the network element operating system process(es) 1012Adetermines the correct type of device driver and installs this devicedriver in the host operating system 1014 kernel 1028. In one embodiment,the network element operating system process(es) 1012A includes avariety of device drivers in part of the data of the network elementoperating system container 1010A and selects the appropriate devicedriver to be installed. In another embodiment, the network elementoperating system process(es) 1012A retrieves the appropriate devicedriver from a remote site (e.g., a website of the network elementoperating system process(es) 1012A vendor, or the host operating system1014 vendor, or another remote site).

In one embodiment, instead having one container for the network elementoperating system, the network element can have the functionality of thenetwork element operating system partitioned into multiple containerswith multiple processes, where one or more of the processes probes thehardware present in data plane and installs the appropriate device driveinto the kernel of the host operating system. FIG. 11 is a block diagramof one embodiment of a process to install a device driver into a kernelof a host operating system by a network operating system functionsexecuting in multiple containers. In FIG. 11, network element 1100includes a control plane 1102 coupled to a data plane 1104. In oneembodiment, the data plane 1104 includes one or more hardware forwardingengines 1116 and one or more hardware tables 1118. In one embodiment,the hardware forwarding engines 1116 are the hardware forwarding enginesas described in FIG. 1 above. Furthermore, in this embodiment, thehardware tables 1118 are the hardware tables as described in FIG. 1above. In one embodiment, the data plane 1104 processes the incomingdata 1120 using the hardware forwarding engines 1116 according to thedata in the hardware tables 1118. In this embodiment, the hardwareforwarding engines 1116 decide whether to forward the data to a nexthop, forward the data to the control plane 1102, and/or further processthe data. If the hardware forwarding engine 1116 determines that theincoming data is control plane data, the hardware forwarding engine 1116forwards this control plane data 1122 to the network element operatingsystem process(es) 1112A, where the network element operating systemprocess(es) 1112A processes this control plane data 1122. If theincoming data 2110 is data plane data, the hardware forwarding engines1116 can determine the next hop for this data and/or perform furtherprocessing of this data (e.g., apply a quality service, perform trafficshaping, process with relevant ACLs, and/or other processing).

The control plane 1102, in one embodiment, includes a CPU 1106 andmemory 1108. In this embodiment, the memory 1108 includes a hostoperating system 1114 and containers 1110A-N that are instantiated bythe host operating system 1114, another container, or some othermechanism. In one embodiment, a container 1110A-N is a namespaceinstance of an operating system level virtualization. In thisembodiment, the container 1110A-N is used for deploying and runningprocesses without launching an entire virtual machine for each of theseprocesses. Instead, these multiple isolated containers are run on asingle control host operating system 1114 and access the kernel of thehost operating system 1114. Because these containers 1110A-N share thesame host operating system kernel as the network element 1100, thecontainers can be more efficient than virtual machines, which requireseparate operating system instances. The containers 1110A-N include thecomponents used to run the desired software, such as files,environmental variables, data, metadata, and/or libraries. In oneembodiment, the containers 1110A-N and/or 1130 are compatible with theDOCKER types of containers.

As described above, the memory 1108 includes the network elementoperating system containers 1110A-N and further can include athird-party container 1130 with a third-party process 1132 that executeswithin the third-party container 1130. In one embodiment, thefunctionalities of the network element operating system are split intomultiple containers 1110A-N, where there one or more network elementoperating system process(es) 1112A-N per container 1110A-N. In thisembodiment, each of the network element operating system containers1110A-N is an isolated namespace used to execute the network elementoperating system process(es) 1112A-N, where the network elementoperating system process(es) 1112A-N can be used to manage the datastored in the hardware tables 1118 and/or other managementfunctionalities of the network element 1100. By executing the networkelement operating system process(es) 1112A in the network elementoperating system containers 1110A-N, a network element operating systemcollectively can be run on a network element that is from a differentvendor than the one that produced the network element operating system.For example and in one embodiment, a network element operating systemfrom one vendor can be run on a white box network element that has adifferent vendor. Thus, instead of a network element operating systemacting as the host operating system for the network element 1100, thenetwork element operating system acts as an application executing on thenetwork element 1100.

In one embodiment, one of the containers 1110A-N is used to execute adatabase process that receives and stores state data of the networkelement operating system processes 1112A-N. in this embodiment, thedatabase process is used to store various state data that is furtherused by the control plane to manage the network element 1100.

In a further embodiment, each of the network element operating systemprocesses 1112A-N can share libraries from one or more layers for imagecompatibility and reduced image size. In this embodiment, theselibraries (were other types of binaries) our shared even when thenetwork operating system is partitioned among multiple containers1110A-N. For example and in one embodiment, a routing container willhave layers, such as a routing layer where a routing process executes,and a basic layer that is utilized by the routing layer. In a furtherexample, hardware container will have the layers of the hardware agentsand the basic layer. In these examples, the binary for the basic layerused by the routing and the hardware containers is the same binary,which leads to a reduced image size for these network element operatingsystem processes it is also used for the image compose ability for theseprocesses.

In a further embodiment, these multiple container 1110A-N instances areused to manage different aspects of the network element. For example andin one embodiment, there can be processes 1112A-N for the quality ofservice, access control lists management (or other types of security),policy service, fan agent, light emitting diode agent, temperaturesensor agent, database service, management service(s), processes tosupport networking protocols (e.g. STP, routing protocols (e.g. such asRIP, BGP, OSPF, IS-IS, IGRP, EIGRP, PIM, DVMRP, and any/or other type orunicast or multicast routing protocol), MPLS, and/or other types ofnetworking protocols), network flow management applications (e.g.,openflow, directflow), process manager, and/or other types of processesfor other types of functionality of the network element 1100.

In one embodiment, the one or more of the network element operatingsystem process(es) 1012A-N programs the hardware tables 1118 using aninstall device driver 1126 that is part of kernel 1128 of the hostoperating system 1114. In this embodiment, because the host operatingsystem 1114 does not know ahead of time which type of network elementoperating system process(es) 1112A-N will be instantiated in the controlplane 1102, the one or more of the network element operating systemprocess(es) 1112A-N installs the device driver 1126 in the kernel 1128based on the type of hardware in the data plane 1104. In thisembodiment, one or more of the network element operating systemprocess(es) 1112A-N determines the type of hardware in the data plane1104 by invoking a utility of the host operating system to return systeminformation as described in FIG. 10 above.

In one embodiment, a network element operating system container 1110Bincludes a hardware agent (e.g., network element operating systemprocess(es) 1112B). In this embodiment, the hardware agent 1112B candetect the type of hardware forwarding engine 1116 is in the data plane1104 and retrieve the corresponding device driver 1126 for the dataplane 1104. In addition, the hardware agent 1112B can further installthe device driver 1126 into the host operating system kernel 1128. Forexample and in one embodiment, the hardware forwarding engine 1116 canbe a certain type of ASIC that is used in the data plane 1104. Based onthe type of ASIC, the hardware agent 1112B determines the correct typeof device driver and installs this device driver in the host operatingsystem 1114 kernel 1128. In one embodiment, the network elementoperating system process(es) 1112A includes a variety of device driversin part of the data of the 1112B and selects the appropriate devicedriver to be installed. In another embodiment, the 1112B retrieves theappropriate device driver from a remote site (e.g., a website of thenetwork element operating system process(es) 1112A vendor, or the hostoperating system 1114 vendor, or another remote site).

FIG. 12 is a flow diagram of one embodiment of a process 1200 to installa device driver into a kernel of a host operating system by a networkoperating system executing in a container. In FIG. 12, process 1200begins by detecting the hardware in the data plane by a network elementoperating system process in a container at block 1202. In oneembodiment, the network element operating system process is executing ina single container used for the network element operating systemprocesses or can be in a container that is specific for the networkelement operating system process, where there are other containers forother network element operating system processes. In a furtherembodiment, the network element operating system process includes ahardware agent that is executing in a single or one of multiplecontainers for the network element operating system. At block 1204,process 1200 determines the device driver for the hardware. In oneembodiment, process 1200 invoke a utility of the host operating systemto return system information as described in FIG. 10 above. Process 1200retrieves and installs a device driver in the kernel of the hostoperating system at block 1206. In one embodiment, process 1200retrieves the corresponding device driver for the detected hardware froma set of device drivers locally stored or can retrieve the device driverfrom a remote site (e.g., a vendor website or some other remote site).With the retrieved device driver, process 1200 installs the devicedriver in the kernel of the host operating system. At block 1208,process 1200 manages the data stored in the hardware tables using theinstall device driver.

Simulating a Network Topology Using Containers

A network element manufacturer will typically test the network elementsthat the manufacturer produces in a variety of different scenarios andtopologies. Typically, the manufacturer will physically wire up thenetwork elements in the topology under test, configure each of thenetwork elements, and run the test. A downside of physically wiring thenetwork elements is that it is expensive because a set of physicalnetwork elements is required to run the test and is time consuming towire, check for errors, and start the test. Instead of physicallywiring, the manufacturer can simulate a topology of network elementsusing software. FIG. 13 is a block diagram of one embodiment of atopology 1300 of network elements. In FIG. 13, system 1300 illustrates atypology of network elements 1302A-D that can be simulated usingmultiple containers. In one embodiment, the network elements 1302A-D arearranged in a full mesh topology where each of the network elements1302A-D are connected to each of the other network elements 1302A-D.With four network elements 1302A-D illustrated, each of the networkelements 1302A-D will be connected to three other network elements1302A-D. While in one embodiment, there are four network elements1302A-D illustrated in alternate embodiments there can be more or lessnetwork elements in the topology and these network elements can bearranged in a different type of topology for the test (e.g., spine-leaf,ring, dual-ring, star, bus, tree, and/or various other types oftopologies for network elements).

FIG. 14 is a block diagram of one embodiment of a device 1400 simulatingmultiple network elements in multiple containers. In FIG. 14, thetesting device 1400 includes CPU 1402 and memory 1404. In oneembodiment, the CPU 1402 includes one or more processors (and/orprocessing cores) that is used to process instructions for the testingdevice 1400. In addition, the memory 1404 is system memory (e.g., DRAMor some other type of system memory) that is used to store instructionsfor a host operating system 1410, network element containers 1406A-N,and controller container 1412. In one embodiment, the host operatingsystem 1410 is used to instantiate the network element containers1406A-N and controller container 1412. In another embodiment, thenetwork containers 1406A-N can each be alternatively instantiated byanother container or some other mechanism. In one embodiment, device1400 is any type of device that can communicate network data withanother device (e.g., a personal computer, laptop, server, mobile device(e.g., phone, smartphone, personal gaming device, etc.), another networkelement, etc.). In one embodiment, the device 1400 can be a virtualmachine or can be a device that hosts one or more virtual machines.

The network element containers 1406A-N are containers that are used toisolate the network element process(es) 1408A-N within separate namespaces of the containers 1406A-N. In this embodiment, each of thenetwork element process(es) 1408A-N represents one or more of thenetwork elements under test, where the processes 1406A-N are configuredfor the appropriate network topology and configuration. For example inone embodiment, these processes 1406A-N can be configured in a mesh,star, spine-leaf, or other type of network topology. In addition, eachof the processes 1406A-N are configured with the appropriate networkservices and protocols for the test. Alternatively, the services andprotocols can be configured using the same and/or different topologies.In one embodiment, the controller container 1412 includes a controllerprocess(es) 1414 that is used to configure, run, and gather the resultsof the test.

In one embodiment, by simulating this network topology using multiplecontainers, network topologies can be simulated on a massive scale on asingle virtual machine on a server or in the cloud. In this embodiment,this scale can be accomplished by using containers for the simulatednetwork elements because the containers share the same binaries and/orkernel. In addition, an administrator can change or upgrade a networkelement operating system used by changing a single share binary.Furthermore, customers can test drive a network (e.g., a datacenterfabric) from a vendor easily before buying or deploying the networkelements. Customers can additionally develop and test operationaltooling using this simulated network elements before actual deployingthe physical network elements. In one embodiment, each container allowsa simulated network element to behave like a different network element.In one embodiment, a simulated network element should be isolated byusing a separate server, virtual machine, or container. In thisembodiment, using containers offers the isolation with the least amountof overhead, thus allowing a greater scale of network topology to besimulated. In a further embodiment, the processes in the differentcontainers can communicate with each using virtual wires that isprovided by the host operating system (e.g. Linux Bridges, vSwitches(e.g., Open vSwitch), MacVLANS, single root input/outputvirtualization).

FIG. 15 is a flow diagram of one embodiment of a process 1500 tosimulate multiple network elements in multiple containers. In FIG. 15,process 1500 begins by receiving the network topology and configurationinformation for the test be performed at block 1502. In one embodiment,the network topology and configuration information is information usedto set up the simulated network topology of network elements and toconfigure each of these simulated network elements. At block 1504,process 1500 is instantiates the containers for each of the networkelements to be simulated. In one embodiment, there can be one or morenetwork elements simulated in each of the containers. Process 1500configures each container with the network element process(es) at block1506. In one embodiment, process 1500 configures the processes for thetest to be performed using the desired network topology andconfiguration for each of the simulated network elements. For example inone embodiment, process 1500 can configure the simulated networkelements in a full mesh topology, were each of the N network elements isconnected to the other N−1 network elements, and further configure eachof the simulated network elements with the appropriate network servicesand/or protocols. At block 1508, process 1500 performs the testing. Inone embodiment, each of the simulated network elements generates networkdata received, processed, and forwarded according to the test. At block1510, process 1500 gathers the results. In one embodiment, with thegathered results, process 1500 can present the results and/or store theresults for later viewing.

FIG. 16 shows one example of a data processing system 1600, which may beused with one embodiment of the present invention. For example, thesystem 1600 may be implemented including a network element 100 as shownin FIG. 1. Note that while FIG. 16 illustrates various components of acomputer system, it is not intended to represent any particulararchitecture or manner of interconnecting the components as such detailsare not germane to the present invention. It will also be appreciatedthat network computers and other data processing systems or otherconsumer electronic devices, which have fewer components or perhaps morecomponents, may also be used with the present invention.

As shown in FIG. 16, the computer system 1600, which is a form of a dataprocessing system, includes a bus 1603 which is coupled to amicroprocessor(s) 1605 and a ROM (Read Only Memory) 1607 and volatileRAM 1609 and a non-volatile memory 1611. The microprocessor 1605 mayretrieve the instructions from the memories 1607, 1609, 1611 and executethe instructions to perform operations described above. The bus 1603interconnects these various components together and also interconnectsthese components 1605, 1607, 1609, and 1611 to a display controller anddisplay device 1617 and to peripheral devices such as input/output (I/O)devices which may be mice, keyboards, modems, network interfaces,printers and other devices which are well known in the art. In oneembodiment, the system 1600 includes a plurality of network interfacesof the same or different type (e.g., Ethernet copper interface, Ethernetfiber interfaces, wireless, and/or other types of network interfaces).In this embodiment, the system 1600 can include a forwarding engine toforward network date received on one interface out another interface.

Typically, the input/output devices 1615 are coupled to the systemthrough input/output controllers 1613. The volatile RAM (Random AccessMemory) 1609 is typically implemented as dynamic RAM (DRAM), whichrequires power continually in order to refresh or maintain the data inthe memory.

The mass storage 1611 is typically a magnetic hard drive or a magneticoptical drive or an optical drive or a DVD ROM/RAM or a flash memory orother types of memory systems, which maintains data (e.g. large amountsof data) even after power is removed from the system. Typically, themass storage 1611 will also be a random access memory although this isnot required. While FIG. 16 shows that the mass storage 1611 is a localdevice coupled directly to the rest of the components in the dataprocessing system, it will be appreciated that the present invention mayutilize a non-volatile memory which is remote from the system, such as anetwork storage device which is coupled to the data processing systemthrough a network interface such as a modem, an Ethernet interface or awireless network. The bus 1603 may include one or more buses connectedto each other through various bridges, controllers and/or adapters as iswell known in the art.

Portions of what was described above may be implemented with logiccircuitry such as a dedicated logic circuit or with a microcontroller orother form of processing core that executes program code instructions.Thus, processes taught by the discussion above may be performed withprogram code such as machine-executable instructions that cause amachine that executes these instructions to perform certain functions.In this context, a “machine” may be a machine that converts intermediateform (or “abstract”) instructions into processor specific instructions(e.g., an abstract execution environment such as a “process virtualmachine” (e.g., a Java Virtual Machine), an interpreter, a CommonLanguage Runtime, a high-level language virtual machine, etc.), and/or,electronic circuitry disposed on a semiconductor chip (e.g., “logiccircuitry” implemented with transistors) designed to executeinstructions such as a general-purpose processor and/or aspecial-purpose processor. Processes taught by the discussion above mayalso be performed by (in the alternative to a machine or in combinationwith a machine) electronic circuitry designed to perform the processes(or a portion thereof) without the execution of program code.

The present invention also relates to an apparatus for performing theoperations described herein. This apparatus may be specially constructedfor the required purpose, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), RAMs, EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus.

A machine readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; etc.

An article of manufacture may be used to store program code. An articleof manufacture that stores program code may be embodied as, but is notlimited to, one or more memories (e.g., one or more flash memories,random access memories (static, dynamic or other)), optical disks,CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards or othertype of machine-readable media suitable for storing electronicinstructions. Program code may also be downloaded from a remote computer(e.g., a server) to a requesting computer (e.g., a client) by way ofdata signals embodied in a propagation medium (e.g., via a communicationlink (e.g., a network connection)).

FIG. 17 is a block diagram of one embodiment of an exemplary networkelement 1700 that instantiates a container. In FIG. 17, the midplane1706 couples to the line cards 1702A-N and controller cards 1704A-B.While in one embodiment, the controller cards 1704A-B control theprocessing of the traffic by the line cards 1702A-N, in alternateembodiments, the controller cards 1704A-B, perform the same and/ordifferent functions (e.g., instantiate a container as described in FIGS.2-5 above, upgrade a container as described in FIGS. 6-9 above, installa device driver as described in FIGS. 10-12 above, and/or simulate anetwork topology as described in FIGS. 13-15 above). In one embodiment,the line cards 1702A-N process and forward control plane data to thecontroller cards 1704A-B. It should be understood that the architectureof the network element 1700 illustrated in FIG. 17 is exemplary, anddifferent combinations of cards may be used in other embodiments of theinvention.

The preceding detailed descriptions are presented in terms of algorithmsand symbolic representations of operations on data bits within acomputer memory. These algorithmic descriptions and representations arethe tools used by those skilled in the data processing arts to mosteffectively convey the substance of their work to others skilled in theart. An algorithm is here, and generally, conceived to be aself-consistent sequence of operations leading to a desired result. Theoperations are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be kept in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “receiving,” “detecting,” “determining,” “installing,”“updating,” “storing,” “instantiating,” “bridging,” “processing,”“setting,” “synchronizing,” “deleting,” “stopping,” “managing,”“saving,” “performing,” “configuring,” or the like, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The processes and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the operations described. The required structurefor a variety of these systems will be evident from the descriptionbelow. In addition, the present invention is not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the invention as described herein.

The foregoing discussion merely describes some exemplary embodiments ofthe present invention. One skilled in the art will readily recognizefrom such discussion, the accompanying drawings and the claims thatvarious modifications can be made without departing from the spirit andscope of the invention.

What is claimed is:
 1. A non-transitory machine-readable medium havingexecutable instructions to cause one or more processing units to performa method to process control plane data in a network element, the methodcomprising: receiving control plane data by a network element operatingsystem of the network element, wherein at least a functionality of thenetwork element operating system is executing in a container of thenetwork element, the network element including a data plane with aplurality of hardware tables and the host operating system; andprocessing the control plane data by the network element operatingsystem.
 2. The machine-readable medium of claim 1, further comprising:updating, by the network element operating system, at least one of theplurality of hardware tables with the processed control plane data. 2.The machine-readable medium of claim 1, wherein vendors of the networkelement operating system and the host operating system are different. 3.The machine-readable medium of claim 1, wherein vendors of the networkelement operating system and the network element are different.
 4. Themachine-readable medium of claim 1, wherein the network elementoperating system updates the at least one of the plurality of hardwaretables using a device driver.
 5. The machine-readable medium of claim 1,wherein one of the plurality of hardware tables is selected from thegroup consisting of a routing table, a media access control addresstable, and an access control list.
 6. The machine-readable medium ofclaim 1, wherein the network element further includes a control plane,wherein the network element operating system is part of the controlplane.
 7. The machine-readable medium of claim 1, wherein the networkelement operating system is isolated and the host operating system isaccessible to devices other than the network element.
 8. Themachine-readable medium of claim 1, wherein the host operating system isisolated.
 9. The machine-readable medium of claim 8, wherein the networkelement operating system manages a physical management interface of thenetwork element.
 10. The machine-readable medium of claim 9, furthercomprising: bridging the physical management interface to a virtualinterface associated with the container.
 11. The machine-readable mediumof claim 1, wherein each of the network element operating system and thehost operating system are visible to other devices.
 12. Themachine-readable medium of claim 1, wherein a third-party processexecutes within the container.
 13. A non-transitory machine-readablemedium having executable instructions to cause one or more processingunits to perform a method to process control plane data in a networkelement, the method comprising: receiving control plane data by at leastone of a plurality of processes of a network element operating system ofthe network element, wherein the plurality of processes is executing ina plurality of containers of the network element, the network elementincluding a data plane with a plurality of hardware tables; processingthe control plane data by the at least one of the plurality ofprocesses; and updating, by the at least one of the plurality ofprocesses, at least one of the plurality of hardware tables with theprocessed control plane data.
 14. The machine-readable medium of claim13, wherein vendors of the network element operating system and the hostoperating system are different.
 15. The machine-readable medium of claim13, wherein vendors of the network element operating system and thenetwork element are different.
 16. The machine-readable medium of claim13, wherein each of the plurality of processes is selected for the groupconsisting of a routing agent, a policy agent, fan agent, light emittingdiode agent, temperature sensor agent, database process, managementprocesses, and process manager.
 17. The machine-readable medium of claim13, wherein a routing agent is selected from the group consisting of aBorder Gateway Protocol agent, Open Shortest Path First routing agent,multicast routing agent, and Routing Information Base Agent
 18. Themachine-readable medium of claim 13, wherein the network elementoperating system updates the at least one of the plurality of hardwaretables using a device driver.
 19. The machine-readable medium of claim13, wherein a container is namespace instance of an operating-systemlevel virtualization.
 20. The machine-readable medium of claim 13,wherein a third-party process executes within the container.